Skip to content

fix(compilers/openapi)!: scan for the version key past the cap - #441

Merged
fuad-daoud merged 5 commits into
stack/1-detectionfrom
stack/7-review-fixes
Sep 12, 2026
Merged

fix(compilers/openapi)!: scan for the version key past the cap#441
fuad-daoud merged 5 commits into
stack/1-detectionfrom
stack/7-review-fixes

Conversation

@fuad-daoud

@fuad-daoud fuad-daoud commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Breaking — see the last section. Fixes the defects two adversarial reviews of the whole stack found in stack PR 1, which the per-commit gates could not see.

#420's fix missed malformed JSON

decodeFlowEntries returned (probe, true) whenever the input opened with {, swallowing a mid-stream JSON error. That contract is correct for a cut prefix (the cut always errors) and wrong for a whole document, which the new sniffWhole reuses it for. So a >64 KiB JSON document whose openapi key sat behind a syntax error still reported unrecognized-format — the outcome PR 1 says it replaced.

Correction to an earlier version of this description, from the review on PR 1: this was never the motivating case. Valid JSON — Stripe's spec3.json — worked in PR 1 from the start: the cut prefix yields an empty probe, and sniffWhole then reads the whole document successfully. Only malformed JSON was missed.

The key scan was not top-level-scoped

"openapi": matched at any nesting depth, anywhere in the buffer. On main that was bounded to 64 KiB; PR 1 removed the bound without tightening the predicate. So a foreign document nesting an "openapi" key paid a full parse, and one that also failed to parse could be reported as openapi/undecodable-sourcethis compiler claiming bytes that are not its own, which detect.go's own comments forbid.

declaresProbeKey is now declaresBlockKey (bare name at column 0, unchanged) OR declaresFlowKey, the latter requiring the source to open a flow mapping and matching only at depth 1. The 64 KiB bound was not reinstated.

Both fixes were mutation-checked: swallowing the error again reddens the new flow test; restoring the unscoped scan reddens 7 subtests. How the scan compares with main and with the parse, shape by shape, is the section below.

Detection parsed the whole document to find one key

sniffWhole ran yaml.Unmarshal over everything past the cap, ahead of the loader's size and node budgets and with no context to cancel it. Measured inside Detect: 32 MB cost 805 ms and 333 MB, for a file the node budget then refused anyway; main declines it in 11 ms. The earlier commit on this branch scoped the key search but deliberately did not reinstate a bound.

The scan the guard already ran now reads the value beside the key, which answers the whole question without building a tree:

before after
8 MB 209 ms / 82 MB 1.9 ms / 0 allocations
32 MB 805 ms / 333 MB 4.1 ms / 0 allocations

Reading the value closes two more holes the review found:

  • Key order no longer decides the format. The prefix answered on whichever key it reached and returned, so {"swagger":"2.0", …64 KiB…, "openapi":"3.0.3"} read as swagger@2.0 above the cap and openapi@3.0 below it — one document, two answers, which is exactly the property TestDetect_KeyOrderDoesNotDecideTheFormat claims to pin.
  • A key alone is no longer a declaration. Prose beside the word — openapi: is a format at column 0 in a Markdown file, at any size — is declined in silence instead of claimed and reported under this compiler's parse error. That also fixes the same shape below the cap, which predates the stack: main reports such a file as openapi@is a format.

BREAKING. Detection past the cap no longer emits undecodable-source; it cannot, having read one key rather than the document. A source it names and the loader cannot parse is now the compile's finding, so openapi.Compile turns load.ErrParse into that diagnostic instead of returning a Go error. That conversion is not cosmetic: engine.Run wraps a compiler's Go error in its own, and the CLI maps that to exit 2 — the code it uses for being invoked wrong — so without it a broken spec would be reported as a misuse of morphic. Exit 1 and the diagnostic are preserved end-to-end, and the message now carries the loader's own position instead of a throwaway parse's. A document past the cap that declares the key with no version beside it is declined rather than claimed: a scan cannot tell that from another format's file, and claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid — sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

On the review's test finding. All six mutations the review planted stayed green on this branch. Twelve against the new code — including all six where they still apply — are all caught. Adds the engine.Run case the review asked for, over a generated >64 KiB JSON with the version last: nothing else in the suite reaches that path, the corpus's largest spec being 7 KiB. Adds the cap-boundary cases too, where a broken document is what tells the two readings apart — a document both can read cannot, which is why an off-by-one there survived until now.

What the scan reads, and what it declines

The scan reads less than the parse, and the cap decides which answers, so every shape they disagree on is a document that names one format below 64 KiB and another above it. TestReadings_AgreeExceptWhereDeclared runs both readings over one set of bytes: every row either must agree or carries the reason it does not, asserted in both directions, so a tolerance added to the scan deletes a reason and a shape lost adds a row. The committed corpus cannot see any of this — its largest spec is a few KiB — which is why the table constructs the shapes.

Fixed in the scan, where it claimed or declined something the parser disagrees with (each a regression against main or a false claim, found in review):

  • a leading UTF-8 byte-order mark defeated both arms, so a marked spec compiled at the cap and was unrecognized-format one byte past it;
  • the block scan crossed YAML document boundaries, claiming a key in a second document the compile never parses;
  • openapi:3.1.0 at column 0 counted as a key; YAML reads it as a plain scalar, and the parse refuses that document.

Declared, not chased — the whole list of shapes the scan reads less than the parse, each held to its reason by the table:

  • a root << merge key: following it means resolving an anchor, which means the parse the cap exists to avoid, and the input chooses when to trigger it. main read this shape past the cap; this branch declines it in silence.
  • a block document quoting its top-level key ("openapi": 3.0.0 at column 0): admitting it re-opens the guard for zero-indent JSON, which json.MarshalIndent(v, "", "") really produces. Declined on main too.
  • an unquoted key in flow style, {openapi: 3.1.0, …}: the flow scan is a JSON lexer. Declined on main too.
  • an anchor or a tag before the version, openapi: &v 3.1.0 / openapi: !!str 3.1.0: the scan reads the scalar as written, which is no version. Declined on main too.
  • openapi:3.1.0 past the cap goes from undecodable-source on main to a silent decline — the same class as the openapi: [unterminated loss the breaking section describes, listed here so it is in one place.

Stack 7 of 8. Base stack/6-constraints — review and merge bottom-up. Every commit here passed make gate when it landed, and the full gate was re-run on the top of the stack. Run it as GOTOOLCHAIN=go1.26.3 make gate; this machine's Go 1.27 fails it for reasons unrelated to any change (#431).

🤖 Generated with Claude Code

https://claude.ai/code/session_01SYeBgDsskwnyitLgPGCPn1

fuad-daoud added a commit that referenced this pull request Sep 9, 2026
…for it

Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
@fuad-daoud
fuad-daoud added this pull request to stack #444 September 9, 2026 19:31
fuad-daoud added a commit that referenced this pull request Sep 9, 2026
…for it

Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
fuad-daoud added a commit that referenced this pull request Sep 9, 2026
…for it

Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
fuad-daoud added a commit that referenced this pull request Sep 10, 2026
…for it

Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
fuad-daoud added a commit that referenced this pull request Sep 10, 2026
…for it

Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v

@Wahbeh-Mohammad Wahbeh-Mohammad left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through this against binaries built from origin/main, from stack/6-constraints, and from this head, running morphic validate over generated specs sitting on both sides of the 64 KiB cap. Six things below; the inline comments have the detail.

Four of them (BOM, merge key, document boundaries, and the test that claims size-independence) are one story — scanProbe reads a document differently from decodeYAML, and the cap decides which reading answers — so they want one design rather than four patches. The shape I'd suggest:

Fix in the scan, where it claims or declines something the parser disagrees with:

  • a leading UTF-8 BOM defeats both arms (a regression against main);
  • the block scan crosses YAML document boundaries, so it answers for a document nothing downstream reads;
  • openapi:3.1.0 counts as a key, which YAML says it is not.

Declare, don't chase, where a byte scan genuinely cannot reach:

  • a root << merge key — following it means resolving an anchor, which means the parse the cap exists to avoid, and the input chooses when to trigger it;
  • a block document quoting its top-level key (already in the body);
  • flow style without quotes, {openapi: 3.1.0, ...};
  • an anchor or a tag before the version, openapi: &v 3.1.0 / openapi: !!str 3.1.0.

The declaration is worth more as a test than as prose. A table that runs both readings over the same bytes, every row either "must agree" or carrying the reason it does not, means adding a tolerance deletes a reason and losing one adds a row — neither can happen silently, and prose in a doc comment can do neither.

I prototyped all of it locally: go test ./... green, compilers/openapi statement coverage still 100.0%, and three planted mutations (drop the BOM trim / drop the document bound / let the colon stand unseparated) each redden only their own cases.


The title lands a breaking change unmarked, and names the smaller half of it

fix(compilers/openapi): scope the key scan, keep the read's error carries no !, while the body opens with Breaking and commit 34ebb32 is fix(compilers/openapi)!:. PRs here squash-merge, so the title is the subject that lands on main — the ! has to be on it. Adding one to the current title does not fit:

$ printf '%s (#441)' "fix(compilers/openapi)!: scope the key scan, keep the read's error" | wc -c
73

It also describes commit 1 rather than the merged change: "scope the key scan" survives, but the headline is that detection past the cap now reads one key instead of parsing the document, and that is the breaking half. Suggestion:

$ printf '%s (#441)' "fix(compilers/openapi)!: scan for the version key past the cap" | wc -c
69

(fix(compilers/openapi)!: read the version key without parsing is 68 if you prefer naming the cost rather than the boundary.)

Two sentences in the body are now falsified

  • "every case is at least as good as main, and four are better" — the BOM cases and the merge-key case are each worse than main on this head. Worth re-measuring after the BOM fix and restating with the merge-key loss named.
  • "One deliberate loss, for your judgement" — there are more than one, per the list above. Since the repo asks that a deliberate limitation be stated in the code and in the body somewhere the next reader reaches, the body wants the full list and scanProbe's doc wants a pointer to the test that holds it.

Comment thread compilers/openapi/detect.go
Comment thread compilers/openapi/detect.go
Comment thread compilers/openapi/detect.go Outdated
Comment thread compilers/openapi/detect_scan_test.go Outdated
Comment thread compilers/openapi/detect.go
Comment thread compilers/openapi/openapi.go Outdated
@fuad-daoud fuad-daoud changed the title fix(compilers/openapi): scope the key scan, keep the read's error fix(compilers/openapi)!: scan for the version key past the cap Sep 12, 2026
fuad-daoud added a commit that referenced this pull request Sep 12, 2026
Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
@fuad-daoud

Copy link
Copy Markdown
Collaborator Author

Title is now fix(compilers/openapi)!: scan for the version key past the cap (69 with the number); the 75-character commit subject on the branch is reworded under the cap as well.

The two falsified sentences are gone. The body's new section lists what the scan fixes and the whole of what it declines, each shape against main — the merge key named as the one shape main read and this branch does not — and scanProbe's doc points at TestReadings_AgreeExceptWhereDeclared as the list that is checked.

The "A doc sentence" section went with its commit, which moved down to #437 where the review asked for it.

fuad-daoud added a commit that referenced this pull request Sep 12, 2026
Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
Base automatically changed from stack/6-constraints to stack/1-detection September 12, 2026 13:36
fuad-daoud and others added 5 commits September 12, 2026 16:36
The whole-source read #420 added got two answers wrong, both by reusing
machinery written for a cut prefix.

The flow decoder reports "this is a flow mapping" for anything opening with
`{`, and drops the error that ended its walk. For a cut prefix that is right:
the cut always breaks the token stream, so the error describes the cut and not
the document. For a whole document it hides the document's own break. A JSON
source past the cap whose `openapi` key sits behind a syntax error came back
with a nil error, so Detect saw no failure to report and declined it as an
unrecognized format — the very answer #420 set out to replace, still standing
for every JSON source, which is the style the motivating spec is written in.
The decoder now returns the error that stopped it and treats stopping on the
mapping's own closing delimiter or on the entry cap as no error at all;
sniffPrefix drops it along with the cut that caused it, and sniffWhole keeps
it.

The key scan was widened to the whole source without being scoped to the top
level. Its block arm reads column 0 and always was top-level, but its quoted
arm matched `"openapi":` at any depth, anywhere in the buffer. Bounded to the
first 64 KiB that cost a needless parse; over a whole source it makes a claim,
and a wrong one — another format's document that nests such a key and does not
parse was reported as an undecodable OpenAPI source. Saying nothing about
bytes that are not this compiler's own is the rule detection is built on.

The quoted spelling is how flow style writes every key, so flow structure is
what scopes it: a depth-tracking scan reads the root mapping's own entries and
nothing under them, and a source that opens no mapping at all declares nothing
here. It is a lexer rather than a parser because the case it exists for is a
document broken before the key that names it, where there is no tree to ask. A
block document that quotes its top-level key is no longer seen and is declined
in silence, which is the direction to be wrong in.

A valid document past the cap that declares its version last still compiles,
which is what #420 was about.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SYeBgDsskwnyitLgPGCPn1
Detection parsed whole documents to find one key. sniffWhole ran
yaml.Unmarshal over everything past the 64 KiB cap, before the loader's
size and node budgets and with no context to cancel it: 32 MB cost 805 ms
and 333 MB, for a file the node budget then refused anyway. main declines
that file in 11 ms. #441 scoped the key search but left the parse.

The scan the guard already ran is extended to read the value beside the
key, which answers the whole question without building a tree:

  8 MB   209 ms /  82 MB  ->  1.9 ms / 0 allocations
  32 MB  805 ms / 333 MB  ->  4.1 ms / 0 allocations

Reading the value, rather than only finding the key, closes two more
holes. Key order stopped deciding the format: the prefix answered on
whichever key it reached and returned, so a document declaring both read
as swagger@2.0 above the cap and openapi@3.0 below it — one document, two
answers, which is the property TestDetect_KeyOrderDoesNotDecideTheFormat
claims. And a key alone is no longer a declaration: prose beside the word
(`openapi: is a format` in a Markdown file, at any size) is declined in
silence instead of claimed and reported under this compiler's parse error.

BREAKING: detection past the cap no longer emits undecodable-source. It
cannot: it has read one key, not the document. A source it names and the
loader cannot parse is now the compile's finding, so openapi.Compile turns
load.ErrParse into that diagnostic rather than returning a Go error.
engine.Run wraps a compiler's Go error in its own and the CLI maps that to
exit 2 — the code it uses for being invoked wrong — so without this a
broken spec would be reported as a misuse of morphic. Exit 1 and the
diagnostic are preserved, and the message now carries the loader's own
position instead of a throwaway parse's. A document past the cap that
declares the key with no version beside it is declined rather than
claimed: a scan cannot tell that from another format's file, and
claiming the wrong one of those two is the costlier mistake.

The prefix machinery goes with the parse it existed to avoid —
sniffPrefix, sniffWhole, decodeFlowEntries, wholeLines and the entry cap.

Tests: the six mutations the review planted all stayed green. Twelve
against the new code, including all six where they still apply, are all
caught. Adds the engine.Run case over a generated >64 KiB JSON with the
version last — nothing else in the suite reaches that path, the corpus's
largest spec being 7 KiB — and the cap-boundary cases, where a broken
document is what tells the two readings apart.

Refs #420

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R65r3qXXNM9jNbQu5gGj9v
The commit that replaced the past-the-cap parse with a scan left maxSniffBytes'
comment naming sniffWhole, which the same commit deleted, and describing a
document past the cap as "read whole" when it is now not read as a tree at all.

Says what the constant does: it is where detection stops parsing and scans, and
nothing is declined for being large.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016EHKV7ZYQJJXCPyynTWq4P
The scan and the parse read a document differently, and the cap decides which
answers, so every shape they disagree on names one format below 64 KiB and
another above it. Three were the scan's to fix: a UTF-8 byte-order mark
defeated both arms, so a spec an editor had marked compiled at the cap and
was unrecognized one byte past it — the #420 failure back for a new input
class; the block scan crossed YAML document boundaries, claiming a key in a
document the compile never parses; and it took `openapi:3.1.0`, a plain
scalar, for a key the parser says declares nothing.

The mark is trimmed, both arms are bounded to the first document with the
opening marker left behind so a flow document after `---` reads as one, and
a block entry needs the separated colon YAML needs. The shapes the scan does
not read by design — a root merge key, whose `<<` means resolving an anchor
and so the parse the cap exists to avoid, on an input the source chooses; a
quoted key in block style; an unquoted one in flow style; an anchor or a tag
before the version — are declared in a differential table that runs both
readings over one set of bytes, each declared row against its reason and
asserted in both directions, so a tolerance added deletes a reason and a
shape lost adds a row.

declaresProbeKey is reached only after a parse failed, which only runs at or
below the cap, so its doc described a call site this branch deleted and the
test feeding it input past the cap exercised bytes it cannot be handed; the
one row that was not a duplicate moves to the scoping test and the rest go.
The key-order test is named for the one shape it pins rather than for a
size-independence the branch does not have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T5no6iADeMGgYjsYcV5in
undecodable said the source table exists once a parse has failed. It does
not: the parse that failed is the one that would have built the document,
Compile returns none, and a Source of 0 against the nil document engine.Run
hands on resolves to no path — naming nothing while claiming to. The
loader's own message carries the position, so the diagnostic is sited at
NoSource, as Detect's sibling for the same condition already is.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011T5no6iADeMGgYjsYcV5in
@fuad-daoud
fuad-daoud merged commit 62f2fdd into stack/1-detection Sep 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants